Call Tracking for Online Content Items

ABSTRACT

This specification relates to tracking phone calls related to online content items. In general, one aspect of the subject matter described in this specification can be embodied in methods that include the actions of determining a probability that a call to a phone number resulted from a first interaction of a set of interactions. In general, another aspect of the subject matter described in this specification can be embodied in methods that include the actions of determining a rank score for an online content item, including for each call of a plurality of phone calls to the phone number, attributing the call to an online content interaction of a set of online content interactions; determining, using the rank score, whether to provide the online content item to a client device; and providing the online content item to the client device upon determining to do so using the rank score.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.12/828,158, entitled “CALL TRACKING FOR ONLINE CONTENT ITEMS,” filedJun. 30, 2010, which is incorporated herein by reference in itsentirety.

BACKGROUND

This specification relates to tracking phone calls related to onlinecontent items, such as phone calls made in response to onlineadvertising.

Advertisers provide advertisements in different forms in order toattract consumers. An advertisement (“ad”) is a piece of informationdesigned to be used in whole or in part by a user, such as a particularconsumer. Ads can be provided in electronic form. For example, onlineads can be provided as banner ads on a web page, as ads presented withsearch results, or as ads presented in a mobile application.

Advertisers are interested in how effectively their online campaignsgenerate sales leads. Apart from online purchases, many advertisersclose their business over phone calls or consider phone calls as strongsales leads. For some businesses, customers who pick up the phone andmake a call after viewing an online ad may be ten times more likely tomake a purchase than customers who click on a link.

SUMMARY

In general, one aspect of the subject matter described in thisspecification can be embodied in methods that include the actions ofdetermining a probability that a call to a phone number resulted from afirst interaction of a set of interactions, wherein each interaction ofthe set of interactions is associated with a particular client deviceand a particular time; and associating a value with the firstinteraction based on the determined probability. Other embodiments ofthis aspect include corresponding systems, apparatus, and computerprogram products.

These and other embodiments can optionally include one or more of thefollowing features. Determining the probability includes comparing atime when the call was made to the particular time associated with thefirst interaction; and considering a value measure associated with thefirst interaction. The set of interactions comprises a set of adimpressions of an ad, with each ad impression referring to a delivery ofthe ad to a particular client device at a particular time. The adincludes the phone number. The value measure associated with the firstinteraction is based on the placement position of the ad within contentfor the first ad impression.

Determining the probability includes: determining a first locationassociated with the call; determining a second location associated withthe client device associated with the first interaction; and comparingthe first location and the second location.

The actions further include, for each interaction of the set ofinteractions, determining a probability that the call resulted from theinteraction; and attributing the call to the interaction having thehighest probability. The actions further include, for each interactionof the set of interactions, determining a probability that the callresulted from the interaction; and attributing the call in a fractionalamount to each interaction of the set of interactions based on theprobability that the call resulted from the interaction.

The actions further include for each interaction of the set ofinteractions, determining a probability that the call resulted from theinteraction; and attributing the call to an interaction having adetermined probability that exceeds a threshold and, if no interactionhas a determined probability that exceeds the threshold, attributing thecall in a fractional amount to each interaction of the set ofinteractions based on the probability that the call resulted from theinteraction. The actions further include determining a probability thata call to a phone number was associated with a first keyword.

In general, another aspect of the subject matter described in thisspecification can be embodied in methods that include the actions ofdetermining a rank score for an online content item, wherein the onlinecontent item includes a phone number, and wherein determining the rankscore comprises: for each call of a plurality of phone calls to thephone number, attributing the call to an online content interaction of aset of online content interactions, wherein each online contentinteraction refers to a delivery of the online content item to aparticular client device at a particular time, determining a probabilityof a call using the online content interactions attributed to the phonecalls, wherein the probability estimates the probability that a futureonline content interaction will result in a call to the phone number,and using the probability of a call in determining the rank score forthe online content item; determining, using the rank score, whether toprovide the online content item to a client device; and providing theonline content item to the client device upon determining to do so usingthe rank score. Other embodiments of this aspect include correspondingsystems, apparatus, and computer program products.

These and other embodiments can optionally include one or more of thefollowing features. Using the probability of a call in determining therank score comprises determining a user interaction probability based onthe probability of a call and a predicted click-through rate for theonline content item. Using the probability of a call in determining therank score comprises: obtaining a call bid and a click bid from anonline content provider, wherein the call bid specifies an amount ofmoney the online content provider is willing to pay for a call resultingfrom a delivery of an online content item, and wherein the click bidspecifies an amount of money the online content provider is willing topay for a click on an online content item; determining a call rank scorebased on the probability of a call and the call bid; determining a clickrank score based on a predicted click-through rate for the onlinecontent item and the click bid; and determining the rank score based onthe call rank score and the click rank score.

Determining the rank score further comprises: for each call of theplurality of phone calls to the phone number, determining whether thecall was a good call based on a duration of the call; determining aprobability of a good call using the attributions of the callsdetermined to be good calls; and using the probability of a good call todetermine the rank score. A good call is one having a duration thatexceeds a threshold duration or is within a duration range.

Determining the rank score further comprises: for each call of theplurality of phone calls to the phone number, determining whether thecall was a good call based on detection of one or more keywords spokenduring the call; determining a probability of a good call using theattributions of the calls determined to be good calls; and using theprobability of a good call to determine the rank score.

Determining the rank score further comprises: for each call of theplurality of phone calls to the phone number, obtaining a rating from anonline content provider regarding the quality of the call anddetermining whether the call was a good call based on the rating;determining a probability of a good call using the attributions of thecalls and their respective ratings; and using the probability of a goodcall to determine the rank score.

The online content item is an ad. The actions further include: receivinga request for one or more ads; determining a plurality of candidate adsincluding the ad; and using the rank score to select the ad from thecandidate ads and providing the ad in response to the request. Theactions further comprise: receiving a request for a plurality of ads forpresentation with content; determining a plurality of candidate adsincluding the ad; selecting a plurality of responsive ads including thead and assigning a position to each responsive ad; and using the rankscore to boost the ad's position among the responsive ads and providingthe responsive ads in response to the request.

Particular implementations may realize one or more of the followingadvantages. For example, phone calls made in response to onlineadvertising can be attributed to specific online interactions, and theattribution may be used to improve selection of online content includingthe online advertising. The ability to attribute calls to specificonline interactions may be used in permitting advertisers to increasethe effectiveness of their advertising campaigns by bidding for calls.

The details of one or more implementations are set forth in theaccompanying drawings and the description below. Other features,aspects, and advantages will become apparent from the description, thedrawings, and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of an example online advertising environment.

FIG. 2 shows an example timing diagram including three onlineinteractions and a phone call.

FIG. 3A shows a table illustrating how expected revenue can bedetermined for an online content item.

FIG. 3B shows a table illustrating how expected revenue can bedetermined for an online content item based on a probability of a call.

FIG. 4 is a flow diagram of an example process for attributing a phonecall to an ad impression.

FIG. 5 is a flow diagram of an example process for determining a rankscore of an advertisement.

FIG. 6 is a flow diagram of an example process for serving ads usingrank scores of the ads.

DETAILED DESCRIPTION

Techniques are provided for determining the effectiveness ofadvertisements or other online content items in generating phone calls,and using the determined effectiveness in improving advertisements orother content items, or in providing compensation associated with theadvertisements or other online content items. These techniques mayinvolve including a telephone number in an advertisement or inserting adynamically-generated phone number on an advertiser's site. Two examplesof ways in which an advertisement may result in a phone call are asimple example in which an end user sees an advertisement and calls aphone number included in the advertisement, and a more complex examplein which the end user clicks on an advertisement and then visits theadvertiser's site within a certain time period (e.g., 30 days), finds aphone number on the site, and calls the phone number. In either case,two major challenges associated with determining the effectiveness ofthe advertisement are, first, determining how to relate a particularcall to a particular ad impression/query without using a separate numberfor each keyword or each ad impression, and, second, how to incorporatephone calls into a measurement of an advertisement's quality. Ingeneral, an advertisement that causes a user to make a phone call to anadvertiser is substantially more valuable to the advertiser than anadvertisement that causes the user to click on the advertisement.

Techniques also are provided for correlating calls with specific adimpressions by using a probabilistic model that uses signals such as thearea code of the caller, the time of the call, the time of a searchquery, and past history of calls.

According to these techniques, attribution of calls to specific adimpressions may be accomplished through a probability model. For eachpairing of a call and an ad impression for a given advertising campaign,factors including, for example, an ad impression's quality, a querytime, and a user location, are considered to determine the probabilitythat the call happened because of the particular ad impression. Thisprobability may be used in many ways, including attributing a call tomultiple ad impressions proportionally to the probability that each adimpression resulted in the call, attributing the entire call to the adimpression with the highest probability, or using a threshold-basedapproach that attributes the call entirely to a single ad impression ifthe corresponding probability is greater than a threshold (for example,0.5), and otherwise attributing fractions of the call to multiple adimpressions.

Offline calls resulting from an advertisement may be used to adjust arank score associated with the advertisement. The rank score can beused, for example, to select the advertisement in an auction, ordetermine an order in which the advertisement is presented relative toother advertisements are presented on a display. Typically, a rank scoreis based on a quality score for the advertisement and a bid for aninteraction with the advertisement. A quality score is based on, forexample, the ad's click-through rate, the relevance of the ad text,overall historical keyword performance, a landing page associated withthe ad, and so on. In some implementations, the rank score is based onan estimated revenue figure that can be determined using a quality scoreand a bid.

For example, using the attribution of calls to ad impressions asdiscussed above, a model may be generated to predict the probability(pCall) that a call will result from an ad impression. This probabilitycould be incorporated into the ad's rank score in many ways, includingcomputing a user interaction probability as a combination of pCall andpCTR (the probability of a click) and using that probability instead ofpCTR when computing the rank score, taking a bid for a call (call_bid)which is separate from a bid for url click (max cpc), and computing thead's rank score as, for example, pCTR * max_cpc+pCall * call_bid. In thelatter example, the call bid could be per keyword or a single bid for anentire advertising campaign.

In addition, the probability of a call (pCall) can be improved from justpredicting the probability of a call to predicting the probability of agood call. There are many ways that a good call may be defined,including, for example, by using the duration of the call as a way tojudge if the call was good or bad. For example, all calls longer thanone minute are designated as good calls. As another alternative,advertisers report back the value of each call, these reported valuesare normalized, and the normalized values are used as indications of thequality of the call.

Referring to FIG. 1, the techniques may be implemented in an exampleonline environment 100. The online environment 100 can facilitate theidentification and serving of content items, such as web pages andadvertisements, to users. A computer network 110, such as a local areanetwork (LAN), wide area network (WAN), the Internet, or a combinationthereof, connects advertisers 102 a and 102 b, an advertisementmanagement system 104, publishers 106 a and 106 b, user devices 108 aand 108 b, and a search engine 112. Although only two advertisers (102 aand 102 b), two publishers (106 a and 106 b) and two user devices (108 aand 108 b) are shown, the online environment 100 may include largenumbers (e.g., thousands) of advertisers, publishers and user devices.The network 110 can communicate with other networks, both public andprivate, such as a public switched telephone network (PSTN) 120 and/or acellular network 126.

In some implementations, one or more advertisers 102 a and/or 102 b candirectly, or indirectly, enter, maintain, and track advertisementinformation in the advertising management system 104. The advertisementscan be in the form of graphical advertisements, such as banneradvertisements, text only advertisements, image advertisements, audioadvertisements, video advertisements, advertisements combining one ofmore of these different types of advertisements, or any other type ofelectronic advertisement document 120. The advertisements may alsoinclude embedded information, such as links, meta-information, and/ormachine executable instructions, such as HTML or JavaScript™.

A user device, such as user device 108 a, can submit a request 109 forpage content 111 to a publisher or the search engine 112. In someimplementations, the page content 111 can be provided to the user device108 a in response to the page content request 109. The page content caninclude advertisements provided by the advertisement management system104, or can include executable instructions (e.g., JavaScript™instructions) that can be executed at the user device 108 a to requestadvertisements from the advertisement management system 104. Exampleuser devices 108 include, for example, personal computers, mobilecommunication devices, and television set-top boxes.

In some implementations, a mobile device requests ads from theadvertising management system 104. The request can be from a specificapplication running on the mobile device. The advertising managementsystem 104 provides one or more ads to the mobile device forpresentation according to the specific application.

Advertisements can also be provided from the publishers 106. Forexample, one or more publishers 106 a and/or 106 b can submitadvertisement requests for one or more advertisements to the system 104.The system 104 can respond by sending the advertisements to therequesting publisher 106 a or 106 b for placement on one or more of thepublisher's web properties (e.g., websites and other network-distributedcontent). The advertisements can include embedded links to landing pages(i.e., pages on the advertisers' websites to which a user is directedwhen the user clicks an ad presented on a publisher website). Theadvertisement requests can also include content request information.This information can include, for example, the content itself (e.g.,page or other content document), a category corresponding to the contentor the content request (e.g., arts, business, computers, arts-movies andarts-music), part or all of the content request, content age, contenttype (e.g., text, graphics, video, audio and mixed media), andgeo-location information.

In some implementations, a publisher 106 can combine the requestedcontent with one or more of the advertisements provided by the system104. This combination of content and advertisements can be sent to theuser device 108 that requested the content (e.g., user device 108 a) aspage content 111 for presentation in a viewer (e.g., a web browser orother content display system). The publisher 106 can transmitinformation about the advertisements back to the advertisementmanagement system 104, including information describing how (e.g., inHTML or JavaScript™), when, and/or where the advertisements are to berendered.

Publishers 106 a and 106 b can include general content servers thatreceive requests for content (e.g., articles, discussion threads, music,video, graphics, search results, web page listings and informationfeeds), and retrieve the requested content in response to the requests.For example, content servers related to news content providers,retailers, independent blogs, social network sites, or any otherentities that provide content over the network 110 can be publishers.

Advertisements can also be provided through the use of the search engine112. The search engine 112 can receive queries for search results. Inresponse, the search engine 112 can retrieve relevant search resultsfrom an index of documents (e.g., from an index of web pages).

The search engine 112 can also submit a request for advertisements tothe system 104. The request may include a number of advertisementsdesired. This number may depend on the search results, the amount ofscreen or page space occupied by the search results, or the size andshape of the advertisements, for example. The request for advertisementsmay also include the query (as entered, parsed, or expanded),information based on the query (such as geo-location information,whether the query came from an affiliate, and an identifier of such anaffiliate), and/or information associated with, or based on, the searchresults. Such information may include, for example, identifiers relatedto the search results (e.g., document identifiers), scores related tothe search results (e.g., information retrieval (“IR”) scores), snippetsof text extracted from identified documents (e.g., web pages), full textof identified documents, or feature vectors of identified documents. Insome implementations, IR scores can be computed from, for example, dotproducts of feature vectors corresponding to a query and a document,page rank scores, and/or combinations of IR scores and page rank scores.

The search engine 112 can combine the search results with one or more ofthe advertisements provided by the system 104. This combined informationthen can be forwarded to the user device 108 that requested the contentas the page content 111. The search results can be maintained asdistinct from the advertisements, so as not to confuse the user betweenpaid advertisements and search results.

The advertisers 102, user devices 108, and/or the search engine 112 canalso provide usage information to the advertisement management system104. This usage information can include measured or observed userbehavior related to advertisements that have been served, such as, forexample, whether a conversion or a selection related to an advertisementhas occurred. The advertisement management system 104 performs financialtransactions, such as crediting the publishers 106 and charging theadvertisers 102 based on the usage information. Such usage informationcan also be processed to measure performance metrics, such as aclick-through rate (“CTR”), conversion rate, and call-through rate(“KTR”).

A click-through can occur, for example, when a user of a user deviceselects or “clicks” on a link to a content item returned by thepublisher or the advertising management system. The CTR is a performancemetric that is obtained by dividing the number of users that clicked onthe content item, e.g., a link to a landing page, an advertisement, or asearch result, by the number of times the content item was delivered.For example, if a link to a content item is delivered 100 times, andthree users click on the content item, then the CTR for that contentitem is 3%. Other usage information and/or performance metrics can alsobe used.

A “conversion” occurs, for example, when a user consummates atransaction related to an advertisement that was previously served. Whatconstitutes a conversion may vary from case to case and can bedetermined in a variety of ways. For example, a conversion may occurwhen a user clicks on an advertisement, is referred to the advertiser'sweb page, and consummates a purchase there before leaving that web page.A conversion can also be defined by an advertiser to be anymeasurable/observable user action such as, for example, downloading awhite paper, navigating to at least a given depth of a website, viewingat least a certain number of web pages, spending at least apredetermined amount of time on a website or web page, or registering ona website. In another example, a conversion can include a purchase madeusing a phone (e.g., during a voice call with a salesperson), or adetection of a particular keyword or phrase (e.g., “buy” or “purchase”)spoken during a phone conversation. The detection of a particularkeyword or phrase may be accomplished, for example, by transcribing thephone conversation and searching the text of the transcription Otheractions that constitute a conversion can also be used.

A call-through can occur, for example, when a user calls a telephonenumber associated with an advertisement returned by the publisher or theadvertising management system 104.

In addition to the advertisements being selected based on content suchas a search query or web page content of a publisher, the advertisementscan also be selected from an auction. In some implementations, theadvertisement management system 104 includes an auction process. Anadvertiser 102 may be permitted to select, or bid, an amount theadvertiser is willing to pay for each telephone call to the advertiseras a result of a user viewing an advertisement and calling the telephonenumber associated with the advertiser listed in the advertisement. Forexample, advertisers A, B, and C may respectively select, or bid, acost-per-call of $0.05, $0.07, and $0.10. The amount advertiser A willpay for a call placed to the advertiser A is $0.05, the amountadvertiser B will pay is $0.07, and the amount advertiser C will pay is$0.10. Since advertiser C is willing to pay more per call, this mayresult in an advertisement of advertiser C being provided morefrequently than advertisements of advertisers A and B. Theadvertisements, associated usage data, and bidding parameters describedabove can be stored as advertisement data in an advertisement data store114.

As can be appreciated from the foregoing, the advertising managementsystem 104 permits the serving of advertisements targeted to documentsserved by the publishers 106 and the search engine 112. Additionally,the usage information described above can be used by the advertisementmanagement system 104 to serve higher performing advertisements that aremore likely to elicit a response from users of the user devices 108 aand 108 b.

In some implementations, the serving of the advertisements, such as theadvertisement 120, can be further optimized by tracking whether theadvertisers associated with the advertisements are contacted by userscalling the advertiser.

The environment 100 can include a public switched telephone network(PSTN) 120 coupled to the network 110 by a gateway 125. Telephonedevices 122 a-122 f can communicate over the PSTN 120. The PSTN 120includes switching elements 124 a-124 b for identifying information fromcalls received from telephone devices 122 a-122 f and directing thesecalls to the called party. The PSTN 120 can be, for example, acircuit-switched telephone network.

Telephone devices 122 a-122 f comprise any telecommunication deviceoperable to electronically receive and transmit audio, including voiceand dual-tone multi-frequency (DTMF) data. Generally, a telephone deviceoperates through transmission of electric signals over the PSTN 120 toallow users to communicate. The environment 100 may include othercommunication devices such as cellular phones, IP phones, soft phones,and/or other communication devices that may communicate over the network110 either alone or in conjunction with the PSTN 120 and/or othernetworks. In some implementation, telephone numbers provided by Internetadvertisements direct call initiation requests from the telephone device122 to the switching element 124 and from the switching element to thecalling party.

The switching element 124 comprises any hardware, software, and/orfirmware operable to route calls between devices in the PSTN 120 and theentity associated with the telephone number dialed. For example, theswitching element 124 may receive a request to initiate a call from thetelephone device 122 and based, at least in part, on the request, routethe call to another element in communication with the PSTN 120, such asthe advertisement management system 104.

In addition to the PSTN network 120, other telephone networks and/orprotocols can also be used. For example, a cellular telephone 122 g cancommunicate with the network 110 over a cellular network 126 and agateway 127. Likewise, a voice over Internet Protocol (VoIP) telephone122 h can communicate over the network 110.

In some implementations, the advertisement management system 104 canassociate a telephone number that is displayed on a web-basedadvertisement with an advertiser. The advertisement management system104 can include a call-through engine 130 that may include instructions,algorithms, or other directives for mapping a telephone number displayedin an Internet advertisement to an associated advertiser 102. Forexample, a user of the user device 108 a may call the number displayedon an Internet advertisement by using the telephone 122 a. The switchingelement 124 may receive a request to initiate a call that identifies atelephone number presented in an Internet advertisement, where theidentified telephone number is associated with the advertisementmanagement system 104 and, in response to the request, forward the callto the advertisement management system 104. The call-through engine 130can map the dialed number to the advertiser 102. In connection withidentifying the advertiser 102, the call-through engine 130 may route,switch or otherwise transmit the call in response to the request to theadvertiser 102. Usage and billing data for the telephone call can alsobe collected and updated by the advertisement management system 104 inresponse to the telephone call. A similar call placement and routingprocess can also be used if the user of the user device 108 placed thecall using the cellular telephone 122 g or the VoIP phone 122 h. Theadvertiser, such as the advertiser 102 b, also may track the call bymonitoring received calls and reporting information about the calls tothe advertisement management system 104.

In some implementations, the advertisement management system 104 canpurchase the use of many unique telephone numbers. The advertisementmanagement system 104 can uniquely associate these telephone numberswith the advertisers 102 and display these telephone numbers inadvertisements for the advertisers 102. Accordingly, when the telephonenumbers 134 are called, the calls are routed to the advertisementmanagement system 104 and then routed to the advertisers 102.

In some implementations, a telephone number 134 can include a commontelephone number with many extensions. Therefore, more than oneadvertiser 102 or other entity can be associated with the same commontelephone number. However, each extension is only assigned to oneentity. Therefore, a telephone number 134 that is a common telephonenumber with an extension is unique to the advertiser 102 associated withthat telephone number and extension. For example, the telephone number“888-888-8888” can be associated with more than one advertiser 102; butthe telephone number “888-888-8888, ext. 123” is only associated withone advertiser 102.

In some implementations, the telephone numbers 134 can be used todetermine whether a user contacts the advertiser 102 as a result ofviewing the advertisement provided by the advertising management system.If, for example, a user contacts the advertiser 102 by dialing aparticular telephone number 134, the call-through engine 130 candetermine that the user viewed the particular telephone number 134 onthe web-based advertisement. If, however, a user contacts the advertiserby dialing a different telephone number, it may be concluded that theuser most likely did not see the web-based advertisement that displayedthe particular telephone number 134, or that the advertisement was notthe reason for the call, since the second telephone number is not listedin the advertisement.

In some implementations, users can opt-out of having their phone callslinked to advertisements. For example, after dialing, a user can bepresented with an audible option (e.g., to press a number or speak aword) to opt-out. Other measures to protect user privacy also may beused.

FIG. 2 shows an example timing diagram including three onlineinteractions and a phone call. The timing diagram is discussed toillustrate techniques for determining a probability that a call to aphone number resulted from a given interaction.

Each interaction represents an interaction with an online content item(e.g., a content item provided by advertising management system 104 to auser device 108). The interactions are associated with unique userdevices and unique times. An online content item can be, for example, anadvertisement. An interaction can be, for example, an impression(providing the content item to a user device that presents the contentitem), a user clicking on the content item, or the like.

The online content item is associated with a phone number. For example,the content item can display the phone number, or provide a link to thephone number (e.g., so that a mobile device dials the phone number whenthe link is clicked).

Each interaction has an associated value measure. For example, where auser is presented with multiple content items, the value measure of aparticular content item can be based on the relative position of thecontent item. Where the content item is an advertisement, the valuemeasure can be higher for positions higher on a web page, and the valuemeasure can be higher for positions at the top of the page than forpositions on the right side of the page.

Interaction 1 occurs at t1, interaction 2 occurs at t2, and interaction3 occurs at t3. A phone call is made at tC. The phone call is made tothe phone number. Because interaction 3 occurred after the phone call,the probability that interaction 3 resulted in the phone call can bedetermined to be zero.

For each of interactions 1 and 2, a probability can be determined thatthe interaction resulted in the phone call. Determining the probabilitycan include considering a number of signals, and a classifier can betrained to accept the signals and determine estimated probabilities. Twosignals that may be determined from the example shown in FIG. 2 are thetime of the interaction and the value measure of the interaction.

The time of the interaction can be compared to the time when the callwas made. For example, for interaction 1, the difference between thecall time and the interaction time (tC−t1) can be calculated andcompared to an expected difference. The expected difference can be, forexample, a shorter time difference for certain kinds of content items(e.g., content items where users typically respond quickly, or contentitems have a small amount of text), or a longer time difference forother kinds of content items (e.g., content items having a larger amountof text or typically requiring additional user thought). p The expecteddifference can be provided from a content item provider (e.g., anadvertiser) or can be determined, for example, by training a classifier.If the difference is closer to the expected difference, then theprobability that the call resulted from the interaction is higher.

A higher value measure for an interaction indicates a higher probabilitythat the call resulted from that interaction. For example, becauseinteraction 1 has a higher value measure (V1) than interaction 2 (V2),it is more likely that (on the basis of value measure) that the callresulted from interaction 1 than from interaction 2.

Another example signal is whether the location of the call matches thelocation of the client device associated with an interaction. Thelocation of the call can be estimated using the area code of the phonenumber of the caller or other information (e.g., GPS coordinatesprovided by a mobile phone). The location of the client device can beestimated using, for example, the internet protocol (IP) address of theclient device, or other information (e.g., GPS coordinates provided by amobile phone, a search history associated with the client device, and soon). The closer the two locations are, the higher the probability thatthe call resulted from the interaction.

The call can be attributed to one or both of the interactions. Forexample, if the probability that the call resulted from the firstinteraction is higher than the probability that the call resulted fromthe second interaction, then the call can be attributed exclusively tothe first interaction, or the call can be attributed in a fractionalamount to each interaction. In some implementations, the call isattributed in whole to an interaction if the probability that the callresulted from that interaction is greater than a threshold (e.g.,p>0.75), and otherwise is attributed fractionally.

In some implementations, calls are transcribed and keywords are detectedin the transcription to attribute calls to interactions. For example,where keywords detected in the transcription of a call match keywordspresented in an interaction, the phone call is more likely to haveresulted from that interaction. Thus, if an ad showed a message thatsaid “10% discount just for today,” then the likelihood that a phonecall resulted from the ad is larger when the phone call includes thewords “10% discount” or “discount today.” In some implementations, calltranscription is optional to protect user privacy. For example, the usercan be asked during the call (e.g., by an automated system or asalesperson) for permission to transcribe the call.

Although FIG. 2 illustrates a single phone call and three interactions,an advertising campaign can involve many interactions and many phonecalls resulting from those interactions. For each of the phone calls,the call can be attributed to one or more of the interactions. Using theattributions, a probability of a phone call for a future interaction canbe estimated. For example, for a certain ad provided a number of timesin response to a keyword, the number of resulting calls can be used toestimate the probability that providing the ad in response to thekeyword in the future will result in a call.

The context of an interaction or call can be used to refine an estimatedprobability of a call. By attributing calls to specific interactionswith known contexts, a probability can be estimated for futureinteractions based on the contexts of those future interactions. Thecontext of an interaction is, for example, the time of day theinteraction occurred, whether a click involved in the interaction was a“long” click (e.g., where the user returned to an original web pageafter a threshold amount of time after viewing a web page after aclick), and so on.

For example, by examining the time that an interaction occurred (e.g.,when an ad was served), the probability that a future ad will result ina call can be based on interactions at times that resulted in calls. So,for example, in determining the probability of a call from serving an adat 3:00 PM, the phone calls that resulted when previously serving the adat 3:00 PM can be considered. In another example, by examining a valuemeasure of an interaction (e.g., the placement of an ad on a web page),the probability of a future interaction matching that value measure canbe based on interactions with that value measure that resulted in calls.So, for example, in determining the probability of a call from servingan ad at the top of a web page or on the side of a web page, the phonecalls that resulted from serving the ad at the top or on the side can beconsidered.

The probability that an interaction with an online content item willresult in a phone call can be used to determine a rank score for theonline content item. The rank score can be based, for example, on theexpected revenue from serving the online content item to a clientdevice. In some implementations, the rank score is used to determinewhether to select an online content item, or to boost or decrease anonline content item's position among multiple content items (e.g., tomove the content item, such as an advertisement, up or down a web page,or from the top of a web page to a side of a web page).

In addition to or instead of determining the probability of a call, aprobability of a good call can be determined. A good call can be definedin various ways. In general, a good call is one that is more beneficialto a content provider. For example, when the content provider is anadvertiser, a good call may be a call that results in a sale or a callfrom a customer who is more ready to buy.

Good calls may be identified in a number of ways. For example, a goodcall can be a call that lasts over a certain amount of time (e.g., 1minute), or lasts within a certain range of times (e.g., between 5 and15 minutes). The amount of time can be specified by a content provider,such as an advertiser. In another example, a content provider can reportback a value for each call indicating the quality of the call (e.g., aperson handling the call for the advertiser can enter a number on aphone at the end of the call, or fill out a form on a web site, and soon). The reported values can be used to determine a probability of agood call.

FIG. 3A shows a table illustrating how expected revenue can bedetermined for an online content item. For purposes of illustration, twoads (Ad1 and Ad2) from two different advertisers are discussed. Theadvertisers bid for clicks on their ads. For example, the advertiserscan bid for clicks in response to searches for a certain keyword in asearch engine.

In this example, both advertisers bid $10 for a click. An advertisingsystem determines that the probability of a click (Pclick) on Ad1 is 0.1and the probability of a click on Ad2 is 0.05. These probabilities canbe determined using various techniques, such as, for example, byexamining the click-through rate (CTR) of those ads in the past. Becausethe probability of a click is higher for Ad1, the expected revenue forserving Ad1 is higher ($1 for Ad1 versus $0.50 for Ad2). Due to thehigher expected revenue, Ad1 can be assigned a higher rank score thanAd2, and, in some implementations, Ad1 will be served more frequentlythan Ad2 or in a better position than Ad2.

If the advertiser for Ad2 wants to increase the rank score of Ad2, theadvertiser can increase his bid, or attempt to improve the appearance ofAd2 to increase its probability of being clicked. However, theadvertiser for Ad2 may be more interested in receiving phone calls inresponse to Ad2 than in receiving clicks in response to Ad2.

FIG. 3B shows a table illustrating how expected revenue can bedetermined for an online content item based on a probability of a call.In this example, the advertiser for Ad1 is not bidding on a phone call,and the advertiser for Ad2 is bidding $20 for a phone call. The expectedrevenue for clicks on Ad2 is the same as it was in FIG. 3A (i.e.,$0.50). However, since the probability of a call (Pcall) from servingAd2 is 0.05, the expected revenue for calls from Ad2 is $1 ($20 * 0.05).Thus, when the expected revenue from calls ($1) is combined with theexpected revenue from clicks ($0.50), the total expected revenue is$1.50. Since the total expected revenue for Ad2 ($1.50) is higher thanthe total expected revenue for Ad1 ($1), Ad2 can be assigned a higherrank score than Ad1.

In some implementations, the advertiser's bid for a phone call can bedetermined automatically based on the advertiser's other bids (e.g., theadvertiser's bid for a click). For example, the bid for a call can be afixed amount or percentage above the bid for a click. The amount orpercentage can be based, for example, on keywords associated with theadvertiser's ad campaign.

FIG. 4 is a flow diagram of an example process 400 for attributing aphone call to an ad impression. In some implementations, the process isperformed by an advertising management system (e.g., advertisingmanagement system 104 of FIG. 1). For purposes of illustration, theprocess 400 will be described with respect to a system that performs theprocess 400.

The system generates a number of ad impressions by serving an ad to aplurality of client devices (step 402). Each ad impression is served toa client device at a particular time, and at least some of the adimpressions will be served to unique client devices at unique times.

The system identifies a call to a phone number in the ad (step 404). Insome implementations, the phone number is displayed in the ad. In someother implementations, the ad provides a link to a web site thatincludes the phone number.

For each ad impression, the system determines a probability that thephone call resulted from the ad impression (e.g., that a user saw the adimpression and decided to call the phone number) (step 406). Theprobability is based on one or more factors. For example, the system maycompare the time when the call was made to the time of the ad impressionand a location associated with the call (e.g., based on the area code ofthe calling number) to a location associated with the ad impression(e.g., based the IP address of the client device). The system also mayconsider the position on a web page of the ad impression (e.g., whetherthe ad was displayed on the top of the page or the side of the page).

The system attributes the call to one or more of the ad impressions(step 410). For example, the system may attribute the call to a singlead impression having the highest determined probability, in a fractionalamount to each ad impression based on the determined probability of eachad impression, or to the single ad impression having the highestprobability if that probability exceeds a threshold, and otherwise in afractional amount to each of the ad impressions.

FIG. 5 is a flow diagram of an example process 500 for determining arank score of an advertisement. In some implementations, the process 500is performed by an advertising management system, e.g., advertisingmanagement system 104 of FIG. 1. For purposes of illustration, theprocess 500 will be described with respect to a system that performs theprocess 500.

For a number of phone calls to a phone number provided by an ad, thesystem attributes each phone call to an ad impression (step 502). Insome implementations, the system performs the process 400 described inFIG. 4 to attribute phone calls to ad impressions.

Using the attributions, the system determines a probability of a phonecall (step 504). The probability of a call estimates the probabilitythat a future ad impression will result in a call to the phone number(e.g., that a user will view the ad and decide to call the phonenumber). In some implementations, the system uses the historical callrate for the ad when it has a value measure of the future ad impressionto determine the probability of the phone call.

The system determines a call rank score based on the probability of thecall and a call bid from an advertiser (step 506). The expected revenuefrom calls illustrated in FIG. 3B is an example of a call rank score.

The system determines a click rank score based on the a predictedclick-through rate for the ad and a click bid from the advertiser (step508). The expected revenue from clicks illustrated in FIG. 3B is anexample of a click rank score. The predicted click-through rate istypically determined using historical click-through rates for the ad.

In some implementations, the system determines a probability of a goodcall using attributions of calls determined to be good calls (step 510).For example, the system may determine whether a call is a good callbased on a duration of the call (e.g., whether the call has a durationthat exceeds a threshold duration or is within a duration range). Insome of those implementations, the system determines whether the call isa good call based on detection of one or more keywords (e.g., “buy”)spoken during the call. The system typically transcribes the call todetect the keywords. The system also may obtain a rating from theadvertiser (or an entity associated with the advertiser) regarding thequality of the calls (e.g., a person answering the phone number providesa rating using the phone's keypad or a computer).

The system determines a rank score based on the call rank score and theclick rank score (step 512). The expected revenue from combined callsand clicks illustrated in FIG. 3B is an example of a rank score.

FIG. 6 is a flow diagram of an example process 600 for serving ads usingrank scores of the ads. In some implementations, the process 600 isperformed by an advertising management system, e.g., advertisingmanagement system 104 of FIG. 1. For purposes of illustration, theprocess 600 will be described with respect to a system that performs theprocess 600.

The system receives a request for one or more ads (step 602). The adsare typically for presentation with content, e.g., with search resultsor on a mobile application.

The system determines a plurality of candidate ads (step 604). In someimplementations, the candidate ads are determined by selecting adsassociated with a keyword matching a keyword provided with the requestfor ads.

The system selects responsive ads from the candidate ads using rankscores of the candidate ads (step 606). Typically, the system ranks thecandidate ads and selects a number of the highest ranking ads. In someimplementations, the system determines the rank scores using the process500 illustrated in FIG. 5.

In some implementations, the system assigns positions to the selectedads using the rank scores (step 608). For example, the system assignspositions based on the rank scores of the ads, and the positionsdetermine an order for the presentation of the selected ads (e.g., onthe top or side of a page, in the order on a page, or in a time order tobe displayed in a mobile application). In some implementations, thesystem assigns positions based on factors other than the rank score anduses the rank score to adjust the positions.

The system sends the selected ads (step 610). In some implementations,the system sends the assigned position of the ads or sends the ads inthe order of their assigned positions.

Embodiments of the subject matter and the operations described in thisspecification can be implemented in digital electronic circuitry, or incomputer software, firmware, or hardware, including the structuresdisclosed in this specification and their structural equivalents, or incombinations of one or more of them. Embodiments of the subject matterdescribed in this specification can be implemented as one or morecomputer programs, i.e., one or more modules of computer programinstructions, encoded on computer storage medium for execution by, or tocontrol the operation of, data processing apparatus. Alternatively or inaddition, the program instructions can be encoded on anartificially-generated propagated signal, e.g., a machine-generatedelectrical, optical, or electromagnetic signal, that is generated toencode information for transmission to suitable receiver apparatus forexecution by a data processing apparatus. A computer storage medium canbe, or be included in, a computer-readable storage device, acomputer-readable storage substrate, a random or serial access memoryarray or device, or a combination of one or more of them. Moreover,while a computer storage medium is not a propagated signal, a computerstorage medium can be a source or destination of computer programinstructions encoded in an artificially-generated propagated signal. Thecomputer storage medium can also be, or be included in, one or moreseparate physical components or media (e.g., multiple CDs, disks, orother storage devices).

The operations described in this specification can be implemented asoperations performed by a data processing apparatus on data stored onone or more computer-readable storage devices or received from othersources.

The term “data processing apparatus” encompasses all kinds of apparatus,devices, and machines for processing data, including by way of example aprogrammable processor, a computer, a system on a chip, or multipleones, or combinations, of the foregoing The apparatus can includespecial purpose logic circuitry, e.g., an FPGA (field programmable gatearray) or an ASIC (application-specific integrated circuit). Theapparatus can also include, in addition to hardware, code that createsan execution environment for the computer program in question, e.g.,code that constitutes processor firmware, a protocol stack, a databasemanagement system, an operating system, a cross-platform runtimeenvironment, a virtual machine, or a combination of one or more of them.The apparatus and execution environment can realize various differentcomputing model infrastructures, such as web services, distributedcomputing and grid computing infrastructures.

A computer program (also known as a program, software, softwareapplication, script, or code) can be written in any form of programminglanguage, including compiled or interpreted languages, declarative orprocedural languages, and it can be deployed in any form, including as astand-alone program or as a module, component, subroutine, object, orother unit suitable for use in a computing environment. A computerprogram may, but need not, correspond to a file in a file system. Aprogram can be stored in a portion of a file that holds other programsor data (e.g., one or more scripts stored in a markup languagedocument), in a single file dedicated to the program in question, or inmultiple coordinated files (e.g., files that store one or more modules,sub-programs, or portions of code). A computer program can be deployedto be executed on one computer or on multiple computers that are locatedat one site or distributed across multiple sites and interconnected by acommunication network.

The processes and logic flows described in this specification can beperformed by one or more programmable processors executing one or morecomputer programs to perform actions by operating on input data andgenerating output. The processes and logic flows can also be performedby, and apparatus can also be implemented as, special purpose logiccircuitry, e.g., an FPGA (field programmable gate array) or an ASIC(application-specific integrated circuit).

Processors suitable for the execution of a computer program include, byway of example, both general and special purpose microprocessors, andany one or more processors of any kind of digital computer. Generally, aprocessor will receive instructions and data from a read-only memory ora random access memory or both. The essential elements of a computer area processor for performing actions in accordance with instructions andone or more memory devices for storing instructions and data. Generally,a computer will also include, or be operatively coupled to receive datafrom or transfer data to, or both, one or more mass storage devices forstoring data, e.g., magnetic, magneto-optical disks, or optical disks.However, a computer need not have such devices. Moreover, a computer canbe embedded in another device, e.g., a mobile telephone, a personaldigital assistant (PDA), a mobile audio or video player, a game console,a Global Positioning System (GPS) receiver, or a portable storage device(e.g., a universal serial bus (USB) flash drive), to name just a few.Devices suitable for storing computer program instructions and datainclude all forms of non-volatile memory, media and memory devices,including by way of example semiconductor memory devices, e.g., EPROM,EEPROM, and flash memory devices; magnetic disks, e.g., internal harddisks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROMdisks. The processor and the memory can be supplemented by, orincorporated in, special purpose logic circuitry.

To provide for interaction with a user, embodiments of the subjectmatter described in this specification can be implemented on a computerhaving a display device, e.g., a CRT (cathode ray tube) or LCD (liquidcrystal display) monitor, for displaying information to the user and akeyboard and a pointing device, e.g., a mouse or a trackball, by whichthe user can provide input to the computer. Other kinds of devices canbe used to provide for interaction with a user as well; for example,feedback provided to the user can be any form of sensory feedback, e.g.,visual feedback, auditory feedback, or tactile feedback; and input fromthe user can be received in any form, including acoustic, speech, ortactile input. In addition, a computer can interact with a user bysending documents to and receiving documents from a device that is usedby the user; for example, by sending web pages to a web browser on auser's client device in response to requests received from the webbrowser.

Embodiments of the subject matter described in this specification can beimplemented in a computing system that includes a back-end component,e.g., as a data server, or that includes a middleware component, e.g.,an application server, or that includes a front-end component, e.g., aclient computer having a graphical user interface or a Web browserthrough which a user can interact with an implementation of the subjectmatter described in this specification, or any combination of one ormore such back-end, middleware, or front-end components. The componentsof the system can be interconnected by any form or medium of digitaldata communication, e.g., a communication network. Examples ofcommunication networks include a local area network (“LAN”) and a widearea network (“WAN”), an inter-network (e.g., the Internet), andpeer-to-peer networks (e.g., ad hoc peer-to-peer networks).

The computing system can include clients and servers. A client andserver are generally remote from each other and typically interactthrough a communication network. The relationship of client and serverarises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other. In someembodiments, a server transmits data (e.g., an HTML page) to a clientdevice (e.g., for purposes of displaying data to and receiving userinput from a user interacting with the client device). Data generated atthe client device (e.g., a result of the user interaction) can bereceived from the client device at the server.

While this specification contains many specific implementation details,these should not be construed as limitations on the scope of anyinventions or of what may be claimed, but rather as descriptions offeatures specific to particular embodiments of particular inventions.Certain features that are described in this specification in the contextof separate embodiments can also be implemented in combination in asingle embodiment. Conversely, various features that are described inthe context of a single embodiment can also be implemented in multipleembodiments separately or in any suitable subcombination. Moreover,although features may be described above as acting in certaincombinations and even initially claimed as such, one or more featuresfrom a claimed combination can in some cases be excised from thecombination, and the claimed combination may be directed to asubcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various systemcomponents in the embodiments described above should not be understoodas requiring such separation in all embodiments, and it should beunderstood that the described program components and systems cangenerally be integrated together in a single software product orpackaged into multiple software products.

Thus, particular embodiments of the subject matter have been described.Other embodiments are within the scope of the following claims. In somecases, the actions recited in the claims can be performed in a differentorder and still achieve desirable results. In addition, the processesdepicted in the accompanying figures do not necessarily require theparticular order shown, or sequential order, to achieve desirableresults. In certain implementations, multitasking and parallelprocessing may be advantageous.

1.-20. (canceled)
 21. A computer-implemented method performed by one ormore processors, the method comprising: determining a probability that acall to a phone number resulted from a first interaction of a set ofinteractions, wherein each interaction of the set of interactions isassociated with a particular client device and a particular time; andassociating a value with the first interaction based on the determinedprobability.
 22. The method of claim 21, wherein determining theprobability includes: comparing a time when the call was made to theparticular time associated with the first interaction; and considering avalue measure associated with the first interaction.
 23. The method ofclaim 21, wherein the set of interactions comprises a set of adimpressions of an ad, with each ad impression referring to a delivery ofthe ad to a particular client device at a particular time.
 24. Themethod of claim 23, wherein the ad includes the phone number.
 25. Themethod of claim 23, wherein the value measure associated with the firstinteraction is based on the placement position of the ad within contentfor the first ad impression.
 26. The method of claim 21, whereindetermining the probability includes: determining a first locationassociated with the call; determining a second location associated withthe client device associated with the first interaction; and comparingthe first location and the second location.
 27. The method of claim 21,further comprising: for each interaction of the set of interactions,determining a probability that the call resulted from the interaction;and attributing the call to the interaction having the highestprobability.
 28. The method of claim 21, further comprising: for eachinteraction of the set of interactions, determining a probability thatthe call resulted from the interaction; and attributing the call in afractional amount to each interaction of the set of interactions basedon the probability that the call resulted from the interaction.
 29. Themethod of claim 21, further comprising: for each interaction of the setof interactions, determining a probability that the call resulted fromthe interaction; and attributing the call to an interaction having adetermined probability that exceeds a threshold and, if no interactionhas a determined probability that exceeds the threshold, attributing thecall in a fractional amount to each interaction of the set ofinteractions based on the probability that the call resulted from theinteraction.
 30. The method of claim 21, further comprising determininga probability that a call to a phone number was associated with a firstkeyword.
 31. A system comprising: one or more processors configured tointeract with a computer storage medium in order to perform operationscomprising: determining a probability that a call to a phone numberresulted from a first interaction of a set of interactions, wherein eachinteraction of the set of interactions is associated with a particularclient device and a particular time; and associating a value with thefirst interaction based on the determined probability.
 32. The system ofclaim 31, wherein determining the probability includes: comparing a timewhen the call was made to the particular time associated with the firstinteraction; and considering a value measure associated with the firstinteraction.
 33. The system of claim 31, wherein the set of interactionscomprises a set of ad impressions of an ad, with each ad impressionreferring to a delivery of the ad to a particular client device at aparticular time.
 34. The system of claim 33, wherein the ad includes thephone number.
 35. The system of claim 33, wherein the value measureassociated with the first interaction is based on the placement positionof the ad within content for the first ad impression.
 36. The system ofclaim 31, wherein determining the probability includes: determining afirst location associated with the call; determining a second locationassociated with the client device associated with the first interaction;and comparing the first location and the second location.
 37. The systemof claim 31, the operations further comprising: for each interaction ofthe set of interactions, determining a probability that the callresulted from the interaction; and attributing the call to theinteraction having the highest probability.
 38. The system of claim 31,the operations further comprising: for each interaction of the set ofinteractions, determining a probability that the call resulted from theinteraction; and attributing the call in a fractional amount to eachinteraction of the set of interactions based on the probability that thecall resulted from the interaction.
 39. The system of claim 31, theoperations further comprising: for each interaction of the set ofinteractions, determining a probability that the call resulted from theinteraction; and attributing the call to an interaction having adetermined probability that exceeds a threshold and, if no interactionhas a determined probability that exceeds the threshold, attributing thecall in a fractional amount to each interaction of the set ofinteractions based on the probability that the call resulted from theinteraction.
 40. The system of claim 31, the operations furthercomprising determining a probability that a call to a phone number wasassociated with a first keyword.
 41. A computer storage medium encodedwith a computer program, the program comprising instructions that whenexecuted by data processing apparatus cause the data processingapparatus to perform operations comprising: determining a probabilitythat a call to a phone number resulted from a first interaction of a setof interactions, wherein each interaction of the set of interactions isassociated with a particular client device and a particular time; andassociating a value with the first interaction based on the determinedprobability.